Artificial Intelligence Enabled Metrology

ABSTRACT

Methods and systems for implementing artificial intelligence enabled metrology are disclosed. An example method includes segmenting a first image of structure into one or more classes to form an at least partially segmented image, associating at least one class of the at least partially segmented image with a second image, and performing metrology on the second image based on the association with at least one class of the at least partially segmented image.

This application is a Continuation of U.S. application Ser. No.16/175,013 filed Oct. 30, 2018, which is hereby incorporated byreference.

FIELD OF THE INVENTION

The invention relates generally to artificial intelligence (AI) enabledmetrology, and specifically to AI enabled metrology for use in chargedparticle microscopy.

BACKGROUND OF THE INVENTION

In many areas of industry and research, analysis and measurement ofsmall structures is performed for product/process development, qualitycontrol, medical evaluation, etc. Such analysis and measurement may beperformed using various types of inspection tools, which likely includeforming images of one or more structures of interest. For example, inthe semiconductor industry, charged particle microscopes are used toimage circuit structures on the nanometer scale, which typically becomethe basis for the analysis and measurement tasks. In such an example,measurements are performed on the images themselves to understandpotential for defects and process control. Such analysis andmeasurements, however, require a highly skilled operator to determinewhere to measure and key features for use in performing themeasurements. This may typically be done using the creation of a recipethat can be ran once the key features are identified and located.

This identification and location of the key features by the skilledoperator, however, can be tedious and void of robustness. Additionally,small changes in imaging conditions or manufacturing processes mayrequire manually re-tuning the recipes due to the inability of therecipe to locate the key features on its own. Such requirement tocontinually re-work the recipe due to changes in the imaging and/or themanufacturing makes full automation unreliable and/or unreachable. Inmany instances, the operators are required to screen out false positivesto ensure the accuracy of the analysis. Removal of the constant humaninteraction with the process is desirable in all industries to increaseproductivity and reduce costs. Additionally, a desire for more robustautomatic analysis and measurement of structures, especially smallstructures that experience changes in shape and consistency, is desired.

SUMMARY

Methods and systems for implementing artificial intelligence enabledmetrology are disclosed. An example method includes segmenting a firstimage of structure into one or more classes to form an at leastpartially segmented image, associating at least one class of the atleast partially segmented image with a second image, and performingmetrology on the second image based on the association with at least oneclass of the at least partially segmented image.

Another embodiment includes a charged particle microscope system forperforming metrology on obtained images. The system including an imagingplatform to obtain an one or more images of a part of a sample, each ofthe one or more images including structure. A controller coupled to theimaging platform to at least perform metrology on the structure in atleast one of the images, the controller, coupled to or includingnon-transitory, computer readable medium including code, that whenexecuted by one or more cores, causes the controller to segment a firstimage of the one or more the images of structure into one or moreclasses to form a segmented image, associate at least one class of thesegmented image on with the second image of the one or more images ofthe structure; and perform metrology on the second image of thestructure based on the association of at least one class of thesegmented image.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is an example of a charged particle microscope system inaccordance with an embodiment of the present disclosure.

FIGS. 2A through 2D show example CNNs for segmenting images inaccordance with embodiments of the present disclosure.

FIG. 3 an example method for segmenting an image and performingmetrology using the segmentation to set reference points for metrologypurposes in accordance with an embodiment of the present disclosure.

FIG. 4 is an example method 401 of performing metrology on an image inaccordance with an embodiment of the present disclosure.

FIGS. 5A through 5E show a sequence of example images in accordance withan embodiment of the present disclosure.

FIG. 6 is an example functional block diagram of a computing system 600in accordance with an embodiment of the present disclosure.

Like reference numerals refer to corresponding parts throughout theseveral views of the drawings.

DETAILED DESCRIPTION OF EMBODIMENTS

Embodiments of the present invention relate to AI enhanced metrology. Insome examples, the AI aspect assists placement of metrology-basedanalytical tools on an original image. For example, an input image maybe segmented into one or more classes to determine the location ofreference features, which are then located on the input image and usedas key features as references for metrology processes. However, itshould be understood that the methods described herein are generallyapplicable to a wide range of different AI enhanced metrology, andshould not be considered limiting.

As used in this application and in the claims, the singular forms “a,”“an,” and “the” include the plural forms unless the context clearlydictates otherwise. Additionally, the term “includes” means “comprises.”Further, the term “coupled” does not exclude the presence ofintermediate elements between the coupled items. Additionally, in thefollowing discussion and in the claims, the terms “including” and“comprising” are used in an open-ended fashion, and thus should beinterpreted to mean “including, but not limited to. . . . ” The term“integrated circuit” refers to a set of electronic components and theirinterconnections (internal electrical circuit elements, collectively)that are patterned on the surface of a microchip. The term“semiconductor device” refers generically to an integrated circuit (IC),which may be integral to a semiconductor wafer, separated from a wafer,or packaged for use on a circuit board. The term “FIB” or “focused ionbeam” is used herein to refer to any collimated ion beam, including abeam focused by ion optics and shaped ion beams.

The systems, apparatus, and methods described herein should not beconstrued as limiting in any way. Instead, the present disclosure isdirected toward all novel and non-obvious features and aspects of thevarious disclosed embodiments, alone and in various combinations andsub-combinations with one another. The disclosed systems, methods, andapparatus are not limited to any specific aspect or feature orcombinations thereof, nor do the disclosed systems, methods, andapparatus require that any one or more specific advantages be present orproblems be solved. Any theories of operation are to facilitateexplanation, but the disclosed systems, methods, and apparatus are notlimited to such theories of operation.

Although the operations of some of the disclosed methods are describedin a particular, sequential order for convenient presentation, it shouldbe understood that this manner of description encompasses rearrangement,unless a particular ordering is required by specific language set forthbelow. For example, operations described sequentially may in some casesbe rearranged or performed concurrently. Moreover, for the sake ofsimplicity, the attached figures may not show the various ways in whichthe disclosed systems, methods, and apparatus can be used in conjunctionwith other systems, methods, and apparatus. Additionally, thedescription sometimes uses terms like “produce” and “provide” todescribe the disclosed methods. These terms are high-level abstractionsof the actual operations that are performed. The actual operations thatcorrespond to these terms will vary depending on the particularimplementation and are readily discernible by one of ordinary skill inthe art.

In general, metrology on images, e.g., using images for a basis ofmeasurement, obtained with a charged particle microscope, for example,has conventionally required heavy user interaction to obtain qualitydata. The heavy user interaction may be required due to image contrastmaking edge finding difficult for standard image processing algorithms,and due to deformed structures in the images making structuralrecognition difficult for the image processing algorithms, to provide afew examples. While metrology is a required aspect of process controland defect detection in certain industries, such as the microelectronicsindustry, improvements in image recognition and metrology are greatlydesired irrespective of the industry. It should be noted that while thediscussion herein uses the microelectronics industry to illustrate thedisclosed techniques, the use of the microelectronics industry is notlimiting and the disclosed techniques may be implemented on images ofany kind for any measurement purposes without exceeding the bounds ofthe disclosure, and all current and future uses are contemplated herein.

One solution to the above disclosed problem includes neural networkimage processing to segment images and label some or all pixels of theimage with one or more class designations. The class designation may beused to determine regions or points of interest, e.g., desired featuresfor use in metrology (either measured or for anchoring measurementpoints), of a structure in the original image. Once the region/points ofinterest are segmented, analytical tools, such as active contours,pattern recognition, and boundary location analytics, e.g., edgefinders, may be placed on the image based on the segmentation, forexample, which may then be used to perform metrology or perform themetrology on the structures in the original image. Because thesegmentation into one or more classes is performed by the neuralnetwork, recognition of deformed structures is more easily tolerated andidentified than would be by conventional image processing, to provideone example of improvement.

FIG. 1 is an example of a charged particle microscope system 100 inaccordance with an embodiment of the present disclosure. The chargedparticle microscope (CPM) system 100, or simply system 100, at leastincludes a CPM environment 102, a network 104, one or more servers 106,and an artificial neural network 114. The CPM system 100 may be used toinvestigate and analyze samples of various size and makeup. For oneexample, the CPM system 100 may be implemented, at least partially, atan integrated circuit manufacturing site and used to analyze and measurevarious aspects of wafers and circuits fabricated at the site. In someembodiments, the CPM system 100 may be distributed across variouslocations. For example, the CPM environment 102 may be located at afabrication or development location, the network 104 distributedlocally, regionally, or nationally, and the server 106 located at aserver farm and coupled to the CPM environment 100 via the network 104.Regardless of the organization of the CPM system 100, the system 100 mayat least be used to implement one or more artificial neural networks(ANN) 114 along with one or more analytical algorithms to performvarious metrology-directed tasks.

The CPM environment 102 includes any type of charged particlemicroscope, but the application of the neural network and analyticsdisclosed herein is not limited to charged particle microscopy, which isused for illustrative purposes only. Example CPMs include scanningelectron microscopes (SEMs), transmission electron microscopes (TEMs),scanning transmission electron microscopes (STEMs), focused ion beams(FIBs), and dual beam (DB) systems that include both electron and ionbeam capabilities, to name a few. The CPM environment 102 may be used toobtain electron or ion images of samples, some of which may be thinsections, e.g., lamellae, taken from a larger sample or wafer. The CPMenvironment 102 may include various aspects that can be contained in asingle tool or that may be situated in separate tools. For example, theCPM environment 102 may include an imaging platform 108, e.g., an SEM,TEM, or STEM, a sample preparation platform 110, and one or morecontrollers 112. Of course, each platform 108 and 110 may include morethan one microscope/sample preparation tools as well.

The imaging platform 108 is used to obtain images of samples, some ofthe samples may have been prepared by the sample prep platform 110, butthat is not necessary. The images are obtained using an electron and/orion source to irradiate the sample with a respective beam of chargedparticles. In some examples, the charged particle beam imaging isobtained by a scanned beam, e.g., moved across the sample, while otherexamples the charged particle beam is not scanned. Backscattered,secondary, or transmitted electrons, for example, are then detected andgray scale images formed based thereon. The images include gray scalecontrast depending on the materials of the sample, where the changes ingray scale indicate changes in the material type or crystal orientation.The imaging platform 108 may be controlled by internal controls (notshown), controller 112, or a combination thereof.

The sample prep platform 110 forms some of the samples that are imagedby the imaging platform 108. Of course, imaged samples may also beformed by other tools (not shown). The sample prep 110 may, for example,be a DB system that uses a FIB to prepare and assist in the removal of athin sample from a larger sample, such as by ion milling, ion inducedetching, or a combination thereof, and other processes to process thesample for imaging. Other processes may include, but are not limited to,planarizing mills/etches, fiducial generation, cross-section formation,top-down lamella preparation, etc. The sample prep platform 110 may alsoinclude an electron imaging component that allows the sample prepprocess to be monitored, but the electron imaging component is notrequired. In some embodiments, the sample prep platform 110 may includeother physical preparation aspects—lasers, cutting tools, resinencapsulation tools, cryogenic tools, etc.—that are used to prepare thesample for the imaging platform 108. The sample prep platform 110 may becontrolled by internal controls (not shown), controller 112, or acombination thereof.

The network 104 may be any kind of network for transmitting signalsbetween the CPM environment 102 and the server(s) 106. For example, thenetwork 104 may be a local area network, a large area network, or adistributive network, such as the internet, a telephony backbone, andcombinations thereof.

The servers 106 may include one or more computing platforms, virtualand/or physical, that can run code for various algorithms, neuralnetworks, and analytical suites. While not shown, a user of the CPMenvironment 102 may have access to the servers 106 for retrieval ofdata, updating software code, performing analytical tasks on data, etc.,where the access is through the network 104 from the user's localcomputing environment (not shown). In some embodiments, the useraccesses image data stored on the servers 106, implements segmentationusing the ANN 114 (which may be executed on the servers 106 or the CPMEnvironment 102), and performs metrology at their local computingenvironment.

In operation, one or more images of a sample is obtained by the imagingplatform 108. At least one of the images, which includes one or morestructures of interest for example, may be segmented into one or moreclasses by ANN 114. The ANN 114 may be included with CPM environment102, servers 106, or a combination thereof. The segmented image may thenbe used to locate/identify desired features of each, or at least one, ofthe imaged structures in the one or more acquired images. The desiredfeatures may be located/identified using various techniques, such asoverlaying the segmented image on one of the original images,associating pixels of the desired features in the segmented image withcorresponding pixels in one of the original images, mapping pixels ofthe segmented image onto pixels of like structure on one of the originalimages, to name a few examples. Of course, any technique to providecorrespondence between pixels/features in the segmented image with thesame of the original image is contemplated herein. The located featuresmay then be used as reference points, e.g., boundaries of the structure,apexes of the structure, etc., for metrology purposes. The locatedreference points, which may also be referred to herein as key points,may then be used to facilitate placement of analytical tools, such asboundary locating analytics and active contours for example, which maythen be used for performing metrology on the one or more structures ofthe image. For example, boundary locating analytics, such as an edgefinder algorithm, may be placed on the image based on the location ofthe reference points, which then determines boundaries between differentmaterials/features of the structure. The located boundaries are thenused to guide metrology of at least a portion of the structure. See FIG.5 for an example, which will be discussed in detail below.

In some embodiments, the segmented image, or pixels thereof, isassociated with the image that was segmented. In other embodiments, thesegmented image is associated with one of the other acquired images aslong as the other acquired image is registered with the image used as abasis for the segmentation. For example, a bright field (BF) image and ahigh-angle angular dark field (HAADF) image are acquired simultaneously,where the two images are characterized as being “perfectly registered,”and one of the images, such as the BF image, may be segmented and usedto identify key reference points, which may then be associated withcorresponding points on the HAADF image for metrology purposes. Or, viceversa. To provide another example, two consecutive images of the samesample area may be quickly acquired, and one of the two images used forsegmentation and the other image used for metrology.

While the image provided to the ANN 114 is described as being obtainedby imaging platform 108, in other embodiments, the image may be providedby a different imaging platform and provided to the ANN 114 via thenetwork 104.

In one or more embodiments, the ANN 114, which may also be referred toas a deep learning system, is a machine-learning computing system. TheANN 114 includes a collection of connected units or nodes, which arecalled artificial neurons. Each connection transmits a signal from oneartificial neuron to another. Artificial neurons may be aggregated intolayers. Different layers may perform different kinds of transformationson their inputs.

One type of ANN 114 is a convolutional neural network (CNN). A CNN isconventionally designed to process data that come in the form ofmultiple arrays, such as a color image composed of three two-dimensionalarrays containing pixel intensities in three color channels. Examplearchitecture of a CNN is structured as a series of stages. The first fewstages may be composed of two types of layers: convolutional layers andpooling layers. A convolutional layer applies a convolution operation tothe input, passing the result to the next layer. The convolutionemulates the response of an individual neuron to visual stimuli. Apooling layer combines the outputs of neuron clusters at one layer intoa single neuron in the next layer. For example, max pooling uses themaximum value from each of a cluster of neurons at the prior layer.

In one or more embodiments, the ANN 114 is a CNN configured to detectand/or identify, e.g., classify, objects of interest shown in an inputimage of a sample. An object of interest is a portion of the sample thatis under study. The remaining portions of the specimen provide contextfor the object of interest. However, the object of interest needs to bemeasured while the remaining portions of the specimen may be ignored. Asan example, one or more round or oval structures may be objects ofinterest within an image, and the one or more oval components may bemeasured. Of course, the objects of interest disclosed herein are forillustrative purposes only, and any type of object of interest capturedby a CPM system 100 may be classified by the ANN 114.

Prior to use, the ANN 114 may need to be trained to identify desiredfeatures of structure in an image. Stated another way, the ANN 114 needsto learn how to segment images as desired. The training may typicallyinclude providing the ANN 114 a number of annotated images with theannotations highlighting desired/different components of the structure.For example, boundaries and key reference points may be highlighted. Thetraining images may typically include images of various quality and alsoinclude structure of various conformity with desired shape. Based on thetraining images, the ANN 114 learns how to identify the various classesof any image received regardless of image/structure quality. Further,the amount of training images may be based on the complexity of thestructures being analyzed, with less complexity requiring fewer trainingimages.

FIGS. 2A through 2D show example CNNs 214 for segmenting images inaccordance with embodiments of the present disclosure. The CNNs 214 arenon-limiting examples of the ANN 114, and may be implemented in a CPMsystem, such as system 100. In the disclosed embodiments of FIGS. 2Athrough 2D, one or more CNNs may receive an input, such as an image, andprovide a segmented image as an output. In the embodiments, the CNNs 214segment the input image into one or more classes to provide an outputimage segmented accordingly. It is understood that one or more classesrefer to one or more non-background classes. In the followingdiscussions one class, or at least one class, refers to a non-backgroundclass. The simplest classifier is a binary classifier with at least twoclasses: background, and the object of interest. The one or more classesmay include body of structure, edges of structure, reference points, andcombinations thereof. Of course, fewer or more classes may be used tosegment the input image, and the number of classes implemented maydepend on the complexity of the segmentation along with otherconsiderations. The one or more classes may be used to determineboundaries of a structure in the images, where the boundaries are usedfor references in performing metrology of the structure.

In some embodiments, the segmentation of the image results in everypixel of the image being associated with at least one class, such as abody of structure class, an edges of structure class, and a referencepoints class. In other embodiments, not all pixels may be associated,but instead only pixels of the structure itself may be classified. Forexample, pixels of the image associated with a edge of the structure ora reference point may be identified by the CNN, which may then be usedfor metrology anchoring. Associating only identified pixels of the imagewith a class may result in a partially segmented image, at least interms of representing the segmented image. Of course, other combinationsof classifications may also be implemented.

FIG. 2A shows CNN 214A, which includes two CNNs, CNN ONE and CNN TWO,with each CNN segmenting the input image differently. For example, CNNONE receives the input image and provides a segmented image with thesegmentation directed at one or more classes of the structure. On theother hand, CNN TWO receives the input image and segments for a single,specific class, such as the key points class, and provides an outputimage including the key points class associated with one or more pixelsof the input image. The two output images of CNN 214A may be mapped toor overlaid on the original input image to define the pixels of theoriginal image with the different classes. In some embodiments, however,only the key points image may be overlaid on the original image, wherethe key points will be used as designations where metrology of desiredfeatures of the structure should be anchored/referenced.

FIG. 2B shows an example CNN 214B, which is a single CNN THREE. The CNN214B segments the image into multiple classes and designates each pixelof the image as one or more classes, and provides multiple images asoutputs. For example, one output may be a segmented image that includesall classes except for the key points, whereas a second output mayinclude only the image with the key points classified. Either of bothoutput images may then be mapped to or overlaid on the input image todesignate pixels associated with the anchor/reference points of thestructure.

FIG. 2C shows an example CNN 214C, which includes CNN ONE and CNN TWO.CNN 214C is similar to CNN 214A except for the order of application.Instead of applying CNN ONE and TWO to an image in parallel similar toCNN 214A, CNN 214C applies CNN ONE and TWO in series. For example, CNNONE may segment the image into one or more classes of structure, whileCNN TWO segments the image into the key points class. While the outputimage is shown to only include the key points, the CNN 214C may providean output image that includes all classes.

FIG. 2D shows an example CNN 214D that includes CNN ONE. CNN 214Dsegments an input image into one or more classes and provides an outputimage in response. The output image of CNN 214D may include all classesof the CNN 214, or a subset of the classes. For example, CNN 214D mayprovide a segmented image that includes only a single non-backgroundclass, such as a boundary class or structure class, that may then beused as references for metrology purposes.

FIG. 3 is an example method 301 for segmenting an image and performingmetrology using the segmented image to identify reference points formetrology purposes in accordance with an embodiment of the presentdisclosure. The method 301 may be performed by a CPM system, such assystem 100 for example. Of course, the method may also be performed by acomputing system that is not part of a CPM system, but receives imagesobtained by a CPM system for processing by the method 301. In general,the method 301 may receive or obtain an input image that includes one ormore features, e.g., a TEM image, and provides metrology data of atleast part of one or more of the features in the image or in aregistered image in response.

The method 301 may begin at block 303, which includes obtaining an imageof a structure to measure. The image may be obtained by a CPM system,such as an image platform 108. For example, the image may be a TEM imageof features, e.g., structure, of an integrated circuit. See FIG. 5A forexample. The TEM image may be of a lamella extracted from asemiconductor wafer that includes the features of the integratedcircuit. Of course, other methods may also be used to obtain the image,which also may be of other types of samples, e.g. biological, mineral,crystalline, etc. In some embodiments, the obtained image may be asingle image of the sample, whereas in other embodiments, the image maybe one of multiple registered images of the sample.

Block 303 may be followed by block 305, which includes segmenting theimage of the structure into one or more classes. The segmentation may beperformed by an artificial neural network, such as ANN 114 or CNN214A-D. The image may be segmented into classes to associate pixels ofthe input image with one or more classes, such as body, boundary, keypoints, etc., and the number of classes is not limiting. See FIG. 5B forexample.

Block 305 may be followed by block 307, which includes overlaying atleast one class of the segmented image on the image of the structure toalign the at least one class with the corresponding pixels of the image.For example, pixels of the key points class may be overlaid on theoriginal image to locate features used for anchoring measurementsperformed on the structure in the image. See FIG. 5C for example. Insome embodiments, the key points class may locate and designate boundsor features of the structure where measurements may be anchored, e.g.,started or ended. In some embodiments, the pixels of the key pointsclass may be used to designate pixels in the original image to foranchoring metrology. The designation may be performed by mapping,overlaying or associating the pixels identified as key class to thecorresponding pixels in the original image. As such, the segmented imagemay not need to be overlaid, either actually or virtually, on theoriginal image, but the segmented image is used to identify the pixelsin the original image that serve a good reference anchor point forpurposes of metrology.

In some embodiments, as noted above, the at least one class of thesegmented image may be overlaid on a separate image that is registeredto the original input image. As such, the association of the segmentedimage, or pixels of a desired class of the segmentation, does not needto strictly be made with the image used for the segmentation. As long asthere is registration between two images, then the image used forsegmentation need not be the same image used for metrology. As such, theterm “original image” as used herein does not specifically require theimage used for segmentation but also includes separate, registeredimages of the image used for segmentation.

Block 307 may be followed by block 309, which includes performingmetrology on the original image using the overlaid at least one class ofthe segmented image. The metrology may begin by determining a boundaryof the structure to base the measurements, such as by performing one ormore boundary locating algorithms on the original image using the atleast one class, e.g., key points, as indicating areas where to performthe edge finding algorithms. See FIG. 5D, 5E for example. The boundarylocating algorithms in general use various analytical tools to determinethe location of a boundary, such as active contours, image recognition,and edge finders, to name a few. For example, an edge finder algorithmmay perform a search using grey scale difference or contrast differencesof pixels within an area around the key point associated pixel todetermine a boundary of a feature of interest. The gray scale differencemay be performed in some embodiments by taking the derivative of thegray scale change. In some embodiments, the boundary may be formedbetween two different materials, a metal and an insulator for example.The edge finders may determine the location of key features of theoriginal image using the key features class as an initial guide.Locating the key features may allow further measurements to be made ofthe object of interest.

Block 309 may be followed by block 311, which includes providingmetrology data. The metrology data may include measurements of one ormore areas of the structure, which may be used for process control ordesign validation, to name a couple examples.

In some embodiments, the segmentation of the image along with overlayingthe segmented image, or at least a portion of it, on the original imagemay make the edge finding and metrology more amendable to automation.Additionally, the use of the key points class, for example, may assistthe edge finding algorithms to locate boundaries when the features ofinterest are malformed or when the imaging has experienced problems.Without the segmentation process, a skilled user may have to go throughevery image and locate where the boundaries may be before applying theedge finding algorithm, which is inefficient.

FIG. 4 is an example method 401 of performing metrology on an image inaccordance with an embodiment of the present disclosure. The method 401may be performed by a CPM system, such as system 100 for example. Ofcourse, the method may also be performed by a computing system that isnot part of a CPM system, but receives images obtained by a CPM systemfor processing by the method 401. The method 401 may begin at block 403,which includes locating at least one class of a segmented image that hasbeen overlaid on, mapped to, or associated with an original image. Ingeneral, the segmented image is used to identify pixels in the originalimage that can be used as anchor or reference points for purposes ofperforming metrology. The original image may have been obtained by a CPMsystem, such as system 100, and the segmented image may have beenobtained by a neural network, such as ANN 114 and/or CNN 214A-214D. Atleast some of the pixels of the segmented image may have been assignedone or more classes by the neural network to determine parts of theimage corresponding to the one or more classes. The image, for example,may be a TEM image of circuit structure, which is to be measured. Insome embodiments, the at least one class may be a key points class thatidentifies key features of the structure, which may provide anchorpoints for metrology purposes.

Block 403 may be followed by block 405, which includes placing aboundary locating algorithm, such as an edge finder, on the structure inthe image based on a location of one or more of the at least one classof the segmented image. While an edge finder is used as a specificexample of an analytical tool to determine boundaries, other tools mayalso be implemented, such as active contours or image recognitionalgorithms, and the analytical tool is a non-limiting aspect of thepresent disclosure. The at least one class, the key points for example,may designate which pixels of the original image are indicative ofboundaries of the structure to be measured. Block 405 may be followed byblock 407, which includes performing edge finding of the structure inthe original image based on the location of the edge finder. The edgefinder may begin with forming a virtual area centered on the key pointsclass, then analyzing the pixels in the virtual area to determine theedge of the structure. This determination, while aided by the segmentedimage, is performed separately on the original image to ensure accuracyof the metrology. In some embodiments, a contrast comparison isperformed on the pixels in the virtual area to determine where thestructure boundary is located.

Block 407 may be followed by block 409, which includes measuring desiredaspects of the structure based on the found edges, e.g., boundaries. Forexample, a width of the structure at the location of the key points maybe determined from the image. This process may be performed for multipleareas/segments of the structure to provide metrology data on thestructure—either the entire structure or portions thereof—as show inblock 411.

FIGS. 5A through 5E show a sequence of example images in accordance withan embodiment of the present disclosure. The sequence shows an exampleof a segmented image, overlaying, mapping or associating one class ofthe segmented image to pixels of the original image, and placement ofedge finders based on the identified pixels, which may then be used asthe basis for performing metrology.

FIG. 5A is an example input image 516 in accordance with an embodimentof the present disclosure. The input image 516 may have been obtainedwith a TEM and is an image of a lamella including a cross-section ofcomponents of an integrated circuit. The input image 516 includes aplurality of structures 526 where each structure 526 includes twooval-ish shaped spots surrounded by a dark ring, which is surrounded bya lighter colored material/region. The oval-ish shaped spots may be of afirst material, the dark ring a second material and the surroundinglighter colored area a different material. While the materials may bedetermined using various analytical techniques, EDX for example,knowledge of the materials is unnecessary for the techniques disclosedherein. It should also be noted that in some embodiments, the inputimage itself may be an EDX-generated image, or an electron energy lossspectrum (EELS) image, but the type of image is not limiting. Themeasurements of various features of each structure 526 may be desired.For example, the height and width of the oval-ish shaped spots may bedesired. In other embodiments, the height and width of the columns belowthe spots may also be desired. To determine these measurements, e.g., toperform metrology on the structures, the input image 516 may be providedto one or more neural networks, such as ANN 114 and/or CNNs 214A-D, forsegmentation. As discussed above, the segmented image may be overlaidon, mapped to, or associated with the input image for placement ofanalytical functions, such as edge finders.

FIG. 5B is an example segmented image 518 in accordance with anembodiment of the present disclosure. The segmented image has had one ormore classes assigned to each pixel of the input image 516. For example,a first class 530 has been assigned to pixels associated with theoval-ish shaped spot, a second class 532 has been assigned to pixelsassociated with dark ring surrounding the spot, a third class 534 hasbeen assigned to key points of the structure, and a fourth class 528assigned to the remaining pixels of the image 516. In some embodiments,all four classes may be overlaid on the input image 516 to aidmetrology. In other embodiments, however, only one or two of the classesmay be used to identify pixels of the input image 516 as key featuresfor anchoring measurements of the features. In general, which class orclasses to use may depend on the robustness of the segmentation andwhether at least one of the classes are robust enough to assistplacement of other metrology tools.

FIG. 5C is an example image 520 in accordance with an embodiment of thepresent disclosure. The image 520 is the input image 516 with the thirdclass 534 overlaid thereon, or at least used to identify pixels in theoriginal image to use as anchors/references for metrology purposes. Thethird class in this example includes the key points class ofsegmentation. The third class 534 locates the pixels in which referencefeatures of the dark ring are located in the input image 516. Forexample, the third class 534 locates extrema of the dark ring on thetop, bottom, left and right sides. These locations, as will be seen, areused as references for metrology purposes.

FIG. 5D is an example image 522 in accordance with an embodiment of thepresent disclosure. The image 522 includes the original input image 516with edge finders 536 placed based on the key points class 534. The edgefinders 536 begin with a virtual area formed on the image and centeredon the pixels identified by the third class 534 locations. The edgefinder algorithm then evaluates the image in the virtual area todetermine an edge of the oval-ish shaped spots of the structures 526.While the third class 534 designates where the boundary likely is, theadditional evaluation ensures the accuracy of the metrology. It shouldbe noted, however, that if the segmentation is robust enough, then thesegmentation may be the basis of the metrology instead of the additionaledge finding analytics. However, the metrology may be performed on theoriginal images, in most examples, based on the edge finding analytics.FIG. 5E is a close up image 524 of image 522, and shows in more detailthe placement of the edge finders 536.

Once the edge finders are located and the edge finding algorithm isperformed, metrology of the oval-ish shaped spots may be performed. Forexample, the distance between the left and right, top and bottom extremaof the oval-ish shaped spots may be determined.

The segmentation of the input image 516 by the neural network ensuresefficient placement of the edge finders without having an operator toview each image to determine where the boundaries of the structurelikely are. This may be especially true for misshapen structures or poorquality images that conventional image recognition algorithms may havetrouble processing. While a single image may not provide much troublefor manual analysis, large numbers of image that require processing andmeasuring make the segmentation almost critical in the metrology of theimages.

According to one embodiment, the techniques described herein areimplemented by one or more special-purpose computing devices. Thespecial-purpose computing devices may be hard-wired to perform thetechniques, or may include digital electronic devices such as one ormore application-specific integrated circuits (ASICs), fieldprogrammable gate arrays (FPGAs), or network processing units (NPUs)that are persistently programmed to perform the techniques, or mayinclude one or more general purpose hardware processors or graphicsprocessing units (GPUs) programmed to perform the techniques pursuant toprogram instructions in firmware, memory, other storage, or acombination. Such special-purpose computing devices may also combinecustom hard-wired logic, ASICs, FPGAs, or NPUs with custom programmingto accomplish the techniques. The special-purpose computing devices maybe desktop computer systems, portable computer systems, handhelddevices, networking devices or any other device that incorporateshard-wired and/or program logic to implement the techniques.

For example, FIG. 6 is a block diagram that illustrates a computersystem 600 upon which an embodiment of the invention may be implemented.The computing system 600 may be an example of the computing hardwareincluded with CPM environment 102, such a controller 112, imagingplatform 108, sample preparation platform 110, and/or servers 106.Additionally, computer system 600 may be used to implement the one ormore neural networks disclosed herein, such as ANN 114 and/or CNNs214A-D. Computer system 600 at least includes a bus 640 or othercommunication mechanism for communicating information, and a hardwareprocessor 642 coupled with bus 640 for processing information. Hardwareprocessor 642 may be, for example, a general purpose microprocessor. Thecomputing system 600 may be used to implement the methods and techniquesdisclosed herein, such as methods 301 and 401, and may also be used toobtain images and segment said images with one or more classes.

Computer system 600 also includes a main memory 644, such as a randomaccess memory (RAM) or other dynamic storage device, coupled to bus 640for storing information and instructions to be executed by processor642. Main memory 644 also may be used for storing temporary variables orother intermediate information during execution of instructions to beexecuted by processor 642. Such instructions, when stored innon-transitory storage media accessible to processor 642, rendercomputer system 600 into a special-purpose machine that is customized toperform the operations specified in the instructions.

Computer system 600 further includes a read only memory (ROM) 646 orother static storage device coupled to bus 640 for storing staticinformation and instructions for processor 642. A storage device 648,such as a magnetic disk or optical disk, is provided and coupled to bus640 for storing information and instructions.

Computer system 600 may be coupled via bus 640 to a display 650, such asa cathode ray tube (CRT), for displaying information to a computer user.An input device 652, including alphanumeric and other keys, is coupledto bus 640 for communicating information and command selections toprocessor 642. Another type of user input device is cursor control 654,such as a mouse, a trackball, or cursor direction keys for communicatingdirection information and command selections to processor 642 and forcontrolling cursor movement on display 650. This input device typicallyhas two degrees of freedom in two axes, a first axis (e.g., x) and asecond axis (e.g., y), that allows the device to specify positions in aplane.

Computer system 600 may implement the techniques described herein usingcustomized hard-wired logic, one or more ASICs or FPGAs, firmware and/orprogram logic which in combination with the computer system causes orprograms computer system 600 to be a special-purpose machine. Accordingto one embodiment, the techniques herein are performed by computersystem 600 in response to processor 642 executing one or more sequencesof one or more instructions contained in main memory 644. Suchinstructions may be read into main memory 644 from another storagemedium, such as storage device 648. Execution of the sequences ofinstructions contained in main memory 644 causes processor 642 toperform the process steps described herein. In alternative embodiments,hard-wired circuitry may be used in place of or in combination withsoftware instructions.

The term “storage media” as used herein refers to any non-transitorymedia that store data and/or instructions that cause a machine tooperate in a specific fashion. Such storage media may comprisenon-volatile media and/or volatile media. Non-volatile media includes,for example, optical or magnetic disks, such as storage device 648.Volatile media includes dynamic memory, such as main memory 644. Commonforms of storage media include, for example, a floppy disk, a flexibledisk, hard disk, solid state drive, magnetic tape, or any other magneticdata storage medium, a CD-ROM, any other optical data storage medium,any physical medium with patterns of holes, a RAM, a PROM, and EPROM, aFLASH-EPROM, NVRAM, any other memory chip or cartridge,content-addressable memory (CAM), and ternary content-addressable memory(TCAM).

Storage media is distinct from but may be used in conjunction withtransmission media. Transmission media participates in transferringinformation between storage media. For example, transmission mediaincludes coaxial cables, copper wire and fiber optics, including thewires that comprise bus 640. Transmission media can also take the formof acoustic or light waves, such as those generated during radio-waveand infra-red data communications.

Various forms of media may be involved in carrying one or more sequencesof one or more instructions to processor 642 for execution. For example,the instructions may initially be carried on a magnetic disk or solidstate drive of a remote computer. The remote computer can load theinstructions into its dynamic memory and send the instructions over atelephone line using a modem. A modem local to computer system 600 canreceive the data on the telephone line and use an infra-red transmitterto convert the data to an infra-red signal. An infra-red detector canreceive the data carried in the infra-red signal and appropriatecircuitry can place the data on bus 640. Bus 640 carries the data tomain memory 644, from which processor 642 retrieves and executes theinstructions. The instructions received by main memory 644 mayoptionally be stored on storage device 648 either before or afterexecution by processor 642.

Computer system 600 also includes a communication interface 656 coupledto bus 640. Communication interface 656 provides a two-way datacommunication coupling to a network link 658 that is connected to alocal network 660. For example, communication interface 656 may be anintegrated services digital network (ISDN) card, cable modem, satellitemodem, or a modem to provide a data communication connection to acorresponding type of telephone line. As another example, communicationinterface 656 may be a local area network (LAN) card to provide a datacommunication connection to a compatible LAN. Wireless links may also beimplemented. In any such implementation, communication interface 656sends and receives electrical, electromagnetic or optical signals thatcarry digital data streams representing various types of information.

Network link 658 typically provides data communication through one ormore networks to other data devices. For example, network link 658 mayprovide a connection through local network 660 to a host computer 662 orto data equipment operated by an Internet Service Provider (ISP) 664.ISP 664 in turn provides data communication services through the worldwide packet data communication network now commonly referred to as the“Internet” 666. Local network 660 and Internet 666 both use electrical,electromagnetic or optical signals that carry digital data streams. Thesignals through the various networks and the signals on network link 658and through communication interface 656, which carry the digital data toand from computer system 600, are example forms of transmission media.

Computer system 600 can send messages and receive data, includingprogram code, through the network(s), network link 658 and communicationinterface 656. In the Internet example, a server 668 might transmit arequested code for an application program through Internet 666, ISP 664,local network 660 and communication interface 656.

The received code may be executed by processor 642 as it is received,and/or stored in storage device 648, or other non-volatile storage forlater execution.

In some examples, values, procedures, or apparatuses are referred to as“lowest”, “best”, “minimum,” or the like. It will be appreciated thatsuch descriptions are intended to indicate that a selection among manyused functional alternatives can be made, and such selections need notbe better, smaller, or otherwise preferable to other selections. Inaddition, the values selected may be obtained by numerical or otherapproximate means and may only be an approximation to the theoreticallycorrect/value.

What is claimed is:
 1. A method comprising: segmenting a first image ofstructure into one or more classes to form an at least partiallysegmented image; associating at least one class of the at leastpartially segmented image with a second image; and performing metrologyon the second image based on the association with at least one class ofthe at least partially segmented image.
 2. The method of claim 1,wherein associating at least one class of the at least partiallysegmented image with the second image includes associating on a pixel bypixel basis the at least one class of the segmented image with thesecond image.
 3. The method of claim 2, wherein the at least one classis a key points class that designates key features of structure includedin the second image.
 4. The method of claim 2, wherein performingmetrology on the second image based on the association of at least oneclass of the at least partially segmented image includes placing an edgefinder on the second image based on the location of the at least oneclass.
 5. The method of claim 1, wherein segmenting a first image intoone or more classes includes classifying each pixel of the first imageas belonging to one or more classes of a plurality of classes.
 6. Themethod of claim 5, wherein the plurality of classes comprises astructure body, a structure boundary, and key points, wherein the keypoints indicate key features of the structure for a basis of metrology.7. The method of claim 5, wherein classifying each pixel of the firstimage as belonging to one or more classes of a plurality of classes isperformed by a single convolutional neural network.
 8. The method ofclaim 5, wherein classifying each pixel of the first image as belongingto one or more classes of a plurality of classes comprises: classifyingthe pixels of the input image into a key points class by a firstconvolutional neural network; and classifying the pixels of the inputimage into a remainder of classes of the plurality of classes by asecond convolutional neural network.
 9. The method of claim 1, whereinperforming metrology on the second image based on the association of atleast one class of the segmented image includes placing boundarylocating analytics on the first image based on the location of the atleast one class.
 10. The method of claim 9, wherein the boundarylocating analytics is selected from one of an edge finder algorithm, anactive contour algorithm, and an image recognition algorithm.
 11. Themethod of claim 1, wherein the first and second images are the sameimage.
 12. The method of claim 1, wherein the first and second imagesare separate, registered images of the same structure.
 13. A chargedparticle microscope system for performing metrology on obtained images,the system comprising: an imaging platform to obtain one or more imagesof part of a sample, each of the one or more images including structure;a controller coupled to the imaging platform to at least performmetrology on the structure in at least one of the images, thecontroller, coupled to or including non-transitory, computer readablemedium including code, that when executed by one or more cores, causesthe controller to: segment a first image of the one or more images ofstructure into one or more classes to form a segmented image; associateat least one class of the segmented image with a second image of the oneor more images of the structure; and perform metrology on the secondimage of the structure based on the association of at least one class ofthe segmented image.
 14. The system of claim 13, wherein the code, thatwhen executed, causes the controller to associate at least one class ofthe segmented image with the second image further includes code thatcauses the controller to associate on a pixel by pixel basis the at lastone class of the segmented image with the second image.
 15. The systemof claim 14, wherein the at least one class is a key points class thatdesignates key features of structure included in the first image. 16.The system of claim 13, wherein the code, that when executed, causes thecontroller to segment the first image of structure into one or moreclasses further includes code that causes the controller to classifyeach pixel of the first image of the structure as belonging to one ormore classes of a plurality of classes.
 17. The system of claim 16,wherein the plurality of classes comprises a structure body, a structureboundary, and key points, wherein the key points indicate key featuresof the structure for a basis of metrology.
 18. The system of claim 16,wherein the classification of the first image of the structure asbelonging to one or more classes of a plurality of classes is performedby a convolutional neural network, the convolutional neural networkeither being executed by the controller or a separate computing devicecoupled to the system.
 19. The system of claim 16, wherein the code,that when executed, causes the controller to classify each pixel of thefirst image of the structure as belonging to one or more classes of aplurality of classes further includes code that, when executed, causesthe controller to: classify the pixels of the first image of thestructure into a key points class by a first convolutional neuralnetwork; and classify the pixels of the first image of the structureinto a remainder of classes of the plurality of classes by a secondconvolutional neural network.
 20. The system of claim 13, wherein thecode, that when executed, causes the controller to perform metrology onthe second image of the structure based on the association of at leastone class of the segmented image further includes code that causes thecontroller to place edge finders on the second image based on thelocation of the at least one class.
 21. The system of claim 20, whereinthe code, that when executed, causes the controller to place edgefinders on the second image based on the location of the at least oneclass further includes code that causes the controller to establish anarea on the second image of the structure based on the location of theat least one class, and further includes code to analyze the secondimage of the structure inside the area to determine a boundary of thestructure within the area.
 22. The system of claim 21, wherein thedetermined boundary forms an anchor point for measuring at least aportion of the structure.
 23. The system of claim 13, wherein the firstand second images are the same image.
 24. The system of claim 13,wherein the first and second images are separate, registered images.